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A Method for Rapidly Evaluating the Linearity of Calibration Data 
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Figure 1. Sample Data Set 


A simple technique is presented for determining 
whether or not a set of five data points lies within a 
specified close tolerance of a linear fit. 

The following theorem, which justifies this tech- 
nique, will be proved: Define two arbitrary constants, 

T and C. Select data points (X,-, Y f ) with i = 1,2, 3, 4, 5 
which have the property 



X.-+1 - Xi = C. (1) 
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Figure 2. Difference Plot 


Let f(X) = a 0 + ai X be the least-squares linear fit 
to these data (regression of Y on X). If g(X) = b G + bi X 
is another line which has the property 

|Y, - g(X,)j < T/1.6 for all i, (2) 

then |Y, - f(X,)j < T for all i. (3) 

Consider the following situation. A calibration check 
is run on amplifier modules with a linearity tolerance 
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of 0.25% of the full-scale output. The input signal is 
increased in five equal steps. Full scale output is 
approximately 5 V (tolerance 0.0125 V). The expected 


results follow: 


Coded 

Nominal 

input 

output 

0 

0.00 V 

1 

1.25 V 

2 

2.50 V 

3 

3.75 V 

4 

5.00 V 


A logical first step to verify acceptable data linearity 
is to plot the points and see if a line can be drawn to 
bring the points within tolerance. In this case, how- 
ever, if a graph is plotted using a scale of 1 V : 2 in., 
the tolerance is only 0.025 in., less than 1/32 in. Thus, 
plotted on 8-1/2 x 11 in. graph paper, the tolerance in 
question is barely discernible. To avoid this problem 
of scale, a difference plot is used. 

Figure 1 presents a sample data set (chosen delib- 
erately to exaggerate certain aspects of difference 
plots) in tabular and graphic form, together with an 
approximately fit reference line: Y = 2 + 3X. Figure 
2 shows the difference plot, a graph of the vertical 
separation of each point from the reference line. These 
differences may be plotted on a magnified scale, to 
allow easy display of discrepancies otherwise too small 
to see. The line plotted on Figure 2 is the least-squares 
best fit to the difference data 

h(X) - 1.1 - 0.3X. (4) 

In general, the least-squares line h(X) fitted to a 
difference plot is related to the reference line g(X) 
used to determine the differences and to the least- 
squares line f(X) fitted to the raw data, by the relation- 
ship 

f(X) = g(X) + h(X). (5) 

Thus, in this example, f(X) may be computed from the 
sum 

g(X) = 2.0 + 3. OX 

h(X) - 1.1 — 0.3X 

f(X) = 3.1 4- 2.7X . 

Worst-case analysis was used to obtain the constant 

1.6 which appears in equation 2. Let the tolerance on 
the y-distance be unity. Then the worst case of a line 
within tolerance exists when each of the five points is 
at the maximum distance of ±1 from the reference 
line. There are 32 such cases, but, because of the 
condition of equation 1, imposed on the independent 
variable, certain of the 32 difference patterns possess 
the same maximum absolute distance of a point from 
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the least-squares line fit. Identification of identical 
patterns of differences reduces to eleven the number 
of essentially different cases. Each of these cases is 
tabulated below, giving the absolute value of the 
residual for the farthest point from the least-squares 
line. 

Maximum 
absolute 
residual 

0.0 

1.4 

1.4 

0.8 

1.6 greatest maximum residual 

1.2 

1.2 

1.4 

1.2 

1.4 

1.2 

Since the greatest (absolute) y-distance of one of these 
points from a least-squares line is 1.6, the theorem, 
stated in equations 2 and 3, is proved. Note, however, 
that the theorem provides a sufficient, but not a neces- 
sary condition. 

With this background, the amplifier module cali- . 
bration data may be evaluated. The tolerance is ad- 
justed to 0.0078 V (0.0125/1.6). A standard form is 
used to make difference plots of the calibration data. 

If a line can be drawn on the difference plot which 
passes within 0.0078 V of each data point, the linearity 
requirement is satisfied. If no such line can be drawn, 
judgement is deferred, and the part is held for further 
investigation. 

Notes: 

1. This technique has been successfully applied to 
reduce delays in a large-scale testing program. 
About 95% of a large group of amplifier modules 
which were evaluated using this technique, were 
accepted without waiting for computer curve fits. 

2. No additional documentation is available. Specific 
questions, however, may be directed to: 

Technology Utilization Officer 
Marshall Space Flight Center 
Huntsville, Alabama 35812 
Reference: B70- 10085 
Patent status: 

No patent action is contemplated by NASA. 

Source: Frances A. Norton Bari of 
The Boeing Company 
under contract to 
Marshall Space Flight Center 
(MFS-14834) 
Category 03 
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